The Role of Item Models in Automatic Item Generation

INTRODUCTION Randy Bennett (2001) claimed, a decade ago, that no topic would become more central to the innovation and future practice in educational assessment than computers and the internet. His prediction has proven to be accurate. Educational assessment and computer technology have evolved at a staggering pace since 2001. As a result many educational assessments, which were once given in a paper-and-pencil format, are now administered by computer using the internet. Education Week's 2009 Technology Counts, for example, reported that 27 US states now administer internet-based computerized educational assessments. Many popular and well-known exams in North America such as the Graduate Management Achievement Test (GMAT), the Graduate Record Exam (GRE), the Test of English as a Foreign Language (TOEFL iBT), and the American Institute of Certified Public Accountants Uniform CPA examination (CBT-e), to cite but a few examples, are administered by computer over the internet. Canadian testing agencies are also implementing internet-based computerized assessments. For example, the Medical Council of Canada Qualifying Exam Part I (MCCQE I), which is written by all medical students seeking entry into supervised clinical practice, is administered by computer. Provincial testing agencies in Canada are also making the transition to internet-based assessment. Alberta Education, for instance, will introduce a computer-based assessment for elementary school students in 2011, as part of their Diagnostic Mathematics Program. Internet-based computerized assessment offers many advantages to students and educators compared to more traditional paper-based assessments. For instance, computers support the development of innovative item types and alternative item formats (Sireci & Zenisky, 2006; Zenisky & Sireci, 2002); items on computer-based tests can be scored immediately thereby providing students with instant feedback (Drasgow & Mattern, 2006); computers permit continuous testing and testing on-demand for students (van der Linden & Glas, 2010). But possibly the most important advantage of Item Models 3 computer-based assessment is that it allows educators to measure more complex performances by integrating test items and digital media to substantially increase the types of knowledge, skills, and competencies that can be measured The advent of computer-based testing has also raised new challenges, particularly in the area of item development (Downing & Haladyna, 2006; Schmeiser & Welch, 2006). Large numbers of items are needed to develop the banks necessary for computerized testing because items are continuously administered and, therefore, exposed. As a result, these banks must be frequently replenished to minimize item exposure and maintain test security. Because testing agencies …